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PREFACE 


By resolution dated October 6, 1981, the Board of Trustees of Educational Test- 
ing Service provided for standards for quality and fairness applicable to all ets 
activities. 

BE IT hereby resolved that the Board of Trustees of the Educational Testing 
Service 

1. Adopts as ets policy and approves as ETS Standards for Quality and 
Fairness the principles and policies previously set forth by ets manage- 
ment in its publication entitled Principles, Policies and Procedural 
Guidelines Regarding ETS Products and Services; 

2. Endorses the procedural guidelines developed and published by man- 
agement to implement these principles and policies; 

3. Directs management to continue its program of monitoring adherence 
to the procedural guidelines across ETS programs and services, to 
revise the guidelines as appropriate, and to make an annual report to 
the Board regarding such efforts; 

4. Authorizes management to provide for alternative procedures which 
can accomplish or hold promise of accomplishing the principles and 
policies included in the Standards and to make provision for small, 
new or specialized programs that have special needs; and 

5. Directs the Trustee Committee on Public Responsibility to prepare 
annually an accountability report, to be released publicly, which eval- 
uates actions taken by ETS in accordance with the ETS Standards for 
Quality and Fairness. 

The Standards are designed to ensure that ets products and services meet 
demonstrable criteria with respect to seven areas of basic importance: 
Accountability, Confidentiality of Data, Product Accuracy and Timeliness, 
Research and Development, Tests and Measurement, Test Use, and Technical 
Assistance, Advice, and Instruction. The first three sections of the Standards 
deal with issues that relate to all ETS activities: the responsibilities of ets to 
those affected by its activities; the rights to and limitations on access to data 
collected by ETS; and the control of quality and performance according to 
commitments. The remaining sections concern issues relating to ETS's main 
endeavors: Research and Development, Tests and Measurement, Test Use, 
and Technical Assistance, Advice, and Instruction. 

The Standards are drawn from particular circumstances and needs at ets and 
reflect its objectives. Because of their origin and purposes, the Standards may 
not be useful to organizations whose practices, programs, or services differ 
from those of ETS. 
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The Standards are implemented by ETS management through procedural 
guidelines which provide more detailed and specific guidance for ets's diverse 
programs and services. These procedural guidelines are reviewed and revised 
from time to time by management as new or alternative approaches are 
developed. Management also reviews special applications that are necessary 
to meet the needs of particular programs. 

ETS believes that the Standards contribute significantly to the quality and utility 
of its products and services for clients and consumers. ETS does not have com- 
plete responsibility or authority, of course, to determine how the Standards 
will be implemented in activities for which policy is substantially established 
by a sponsoring group or institution other than ETS. ETS, however, encourages 
and assists those groups and institutions to implement the Standards as their 
activities are related to ETS products and services. 

ETS reviews all its programs regularly to assess compliance with the Standards 
as part of its quality assurance effort. To assist the Trustees to ensure that the 
Standards are interpreted and applied according to their spirit and purpose, a 
Visiting Committee of persons outside ETS who are knowledgeable in the spe- 
cialized aspects of the Standards has been authorized by the Board to review 
periodically the ETS assessment process. Beginning in 1982, the Committee on 
Public Responsibility of the ETS Board of Trustees will issue a public report 
annually on how the organization is meeting these Standards. 

These efforts reflect ets's determination to hold itself accountable to high stan- 
dards of performance in order to safeguard the public interest and the interest 
of individuals using its products and services. 


October 1981 





Gregory R. Anrig 
President 
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ACCOUNTABILITY 


Principle 

£75 acknowledges responsibility for the effective stewardship of its resources 
to the New York Board of Regents which has issued its corporate charter; to 
the governing boards that sponsor and set policy for programs or services in 
which ETS products or services are used; to the individuals and committees 
that advise ETS with respect to appropriate policy for its programs; to the insti- 
tutions and agencies that use ETS products and services; to persons who take 
ETS tests (and parents or guardians of minor persons ), submit data for use by 
ETS or for distribution to others , or participate in research and development 
projects conducted by ETS; and to the professional associations that are con- 
cerned with educational and psychological measurement and research. 


Policies 

A. ETS will furnish appropriate information to those to whom it is responsible so 
they may make informed, independent judgments as to the effectiveness with 
which ETS exercises its stewardship. 

B. ETS will seek, consider and, as appropriate, act on the views of those who 
sponsor, use or are affected by ETS programs and services. 

C. ETS will seek to obtain advice on its activities and policies from qualified men 
and women who are not employed or retained on a regular basis by ETS and 
who are drawn from appropriate professional disciplines, major philosophies 
and points of view, different geographic regions, and the major ethnic groups 
within the relevant population, 

D. ETS will support the activities of professional associations with respect to 
developing and implementing professional standards or codes, making avail- 
able the results of current work, and fostering peer review of its activities. 


Procedural Guidelines 

1. Information should be provided to sponsoring organizations with which ETS 
has contractual relationships in a form that permits evaluation of ETS services 
in terms of: 

a. quality; c. costs; and 

b. timeliness; d. responsiveness to legitimate comments or criticisms. 
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2. Procedures should be established to facilitate communication with sponsors 
by: 

a. meeting at least annually to provide information and to receive comments 
on matters affecting the operations with which they are concerned; 

b. defining a mutually agreeable process to be used to transmit comments 
from sponsors or others and a time period within which the evaluation of 
comments will be completed and reports of actions to be taken by ETS can 
be expected; and 

< making available periodic opportunities for sponsors to express opinions, 
judgments and counsel concerning their activities or programs directly to 
ETS officers not normally responsible for such activities or programs. 

3. Procedures should be established for making available technical and other 
information about ETS products and services to users so that they may eval- 
uate the appropriate use of the product or service and communicate com- 
ments or criticisms to ETS. 

4. Procedures should be established to communicate with, or provide informa- 
tion to, persons who use or take ETS tests, who submit data for use by ETS or 
for distribution to others, or who participate in research and development 
projects conducted by ETS. This information should be communicated by ets 
or the sponsor in such a way that these persons may understand their partici- 
pation with respect to: 

a. the identity and scope of the sponsor's responsibility; 

b. the nature of the product, service or research by which they are affected; 

c . the way in which the product, service or research will likely be used by 
educational institutions or others; and 

d. the channels that have been established for addressing comments or criti- 
cisms to ets or to the sponsor and response thereto. 

5. Organizational and program financial information should be recorded, pro- 
cessed and reported in accordance with generally accepted accounting princi- 
ples and under appropriate safeguards to insure accuracy. 

6. An annual report that provides information about organizational activities and 
finances should be published by ETS on a regular basis and made available to 
any person on request. Program and project reports, including program finan- 
cial information, should be made available in a manner consistent with con- 
tractual understandings. 
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7. Requests for information that is not included in an existing publication should 
be considered by the appropriate sponsor and by ETS. If its disclosure is consis- 
tent with applicable law, with ETS and sponsor policy, and with contractual 
obligations governing confidential or proprietary information, the information 
should be provided. If complying with a request for information results in a 
cost to ETS or a sponsor or affects the normal schedule of fulfilling ets's respon- 
sibilities, ETS may provide the requested information in a reasonable period of 
time and at an appropriate price for the services rendered. Procedures should 
be established, as appropriate, to facilitate responses to these requests. 

8. Changes in federal statutes, regulations and case law that affect research and 
development, testing programs, or advisory and instructional services should 
be monitored to ensure that ETS activities and operations are in compliance as 
relevant federal laws or rules change. Changes in other statutes, regulations 
and case law should be evaluated, as appropriate, for the same purpose. 

9. All proposed new ETS activities should be reviewed by counsel for compliance 
with applicable federal law and state law, as appropriate. ETS officers and staff 
should direct the attention of legal counsel to matters that might affect ETS 
compliance. 

10. Advice should be sought, where appropriate, from men and women drawn 
from diverse backgrounds, interests and experience (e.g., appropriate profes- 
sional disciplines, major philosophies and points of view, various geographic 
regions, and major ethnic, handicapped and other relevant subgroups of the 
population of interest) who are qualified to make a contribution to the direc- 
tion and substance of ETS programs and who are not employed or retained on 
a regular basis by ETS. 

11. Individuals who become members of an ETS external advisory, review or eval- 
uation committee should be informed about the results of the committee's 
work in a reasonable period of time. 

12. A reasonable accommodation should be made with respect to the profes- 
sional responsibilities of the staff in order to permit staff members to attend 
professional meetings, to contribute to the development of professional stan- 
dards or codes, to participate in and benefit from the dissemination of infor- 
mation on subjects of professional interest, and to stay abreast of current con- 
cerns and accomplishments in related fields. 

13. ETS should have effective procedures for peer review whenever it will contrib- 
ute substantially to the quality of ETS work. 

14. ETS should have effective and equitable procedures for handling questions of 
score authenticity arising in connection with the administration of tests. 
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CONFIDENTIALITY 


Principle 

I TS recognizes the right of individuals and institutions to privacy with regard to 
information supplied by and about them that may be stored in data or 
research files held by ETS and the concomitant responsibility to safeguard 
information in its files from unauthorized disclosure. 

Policies 

A. ETS will ask individuals to provide information about themselves only if it is 
potentially useful to those individuals, necessary to facilitate processing of 
data or serves the public interest in improving understanding of human perfor- 
mance. Insofar as possible, individuals should be informed of the purpose for 
which the information is requested. 

B. The right of individuals to privacy regarding information about them that may 
be stored in the data or research files held by ETS extends both to processed 
information, such as scores based on test-item responses, and the raw data on 
which the processed information is based. 

C. ETS will protect the confidentiality of data supplied by institutions or agencies 
about themselves, and so identified, to the extent that such confidentiality 
does not conflict with ets's obligations to individuals. 

D. ETS will not collect or maintain in its data or research files any critical informa- 
tion that in its judgment cannot be protected adequately from improper dis- 
c losure. 

E. ets will encourage the organizations with which it works to adopt policies and 
procedures that adequately protect the confidentiality of the data transferred 
by ETS to those organizations. 


Procedural Guidelines 

1. Information about an individual, which has been identified as such, may not 
be released by ETS to organizations other than those for which the information 
was collected without the consent of that individual. A written exception may 
be made in the case of research studies during which the Committee on Prior 
Review of Research has determined that release of the data serves a public 
need, that there is no satisfactory and reasonable alternative way of obtaining 
the information, that the recipient researcher will use the data in appropriate 
ways and that there are adequate assurances of confidentiality. 
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2. Information about an institution, which has been identified as such, may be 
released from ETS only in a manner consistent with a prior agreement or with 
the consent of the institution or with the approval of the cognizant ETS officer 
and representative of the appropriate sponsor (if any). 

3. An individual should be able, on payment of a reasonable fee, to authorize 
the disclosure of information about himself or herself from program data files 
held by ETS to any appropriate recipient, provided that such authorization is in 
writing and that disclosure is not inconsistent with other ETS or sponsor poli- 
cies and does not violate the privacy of other individuals. Identification of the 
requester, through signature and data file number, or other appropriate meth- 
od, should be required before any such information is released. 

4. In an emergency and when it is to the benefit of the individual, an authoriza- 
tion by telegram or telephone for the release of personal data should be 
acceptable, provided that such authorization includes adequate identifying 
information and that such release is not inconsistent with other ETS or sponsor 
policies. By prior agreement with the individual, authorization by a designated 
agency or institution should also be acceptable. In such instances, the individ- 
ual should be informed that the disclosure has taken place, 

5. If an individual is not competent because of illness or other considerations, 
information about that individual may be released from data files only with 
the consent of the individual's parent or legally appointed guardian. 

6. Unless the access to confidential data can be safeguarded, ETS should not par- 
ticipate in any time-sharing network, data bank, or other electronic data pro- 
cessing or storage system involving units outside ETS. 

7. On submission of appropriate identifying information and payment of a rea- 
sonable fee, an individual should be able to obtain information about himself 
or herself in ETS-held data files for the following purposes: to ascertain the 
accuracy of personal or biographical data and to request verification, within a 
reasonable period of time, of test scores or other processed information from 
tests, questionnaires, or school records, provided such release is consistent 
with sponsor policies. 

8. Procedures should be developed for systematically eliminating from data files 
information that is judged to be out of date and, hence, of minimal value. 

9. Information from ETS-held program data files provided by individuals for a 
designated purpose should not be used or released for another purpose (such 
as a validity study or research project) without the individual's consent except 
when used or released in a form that cannot be identified with the individual. 
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10. ets should refuse to provide personally identifiable information except in 
accordance with these guidelines unless served with a subpoena or other 
court order. In that event, ETS should make appropriate efforts to quash or nar- 
row the subpoena or order or to obtain a protective order to minimize the 
exposure of personally identifiable information. 

11. At the time information is collected and to whatever extent practical, pro- 
grams should inform individuals of the conditions surrounding the release and 
confidentiality of the information about them. 

12. Individuals should be identified in ETS research files only by code numbers. 
Information linking the code numbers to names should be kept in a secure 
location only as long as necessary for purposes such as follow-up studies or 
collating new data, after which the names should be destroyed. 

13. Every organization with which ets works should be informed of the confiden- 
tial nature of any data transferred by ets to that organization or collected by 
the organization on behalf of ets so that appropriate procedures can be 
employed by the recipient organization to protect the confidentiality of such 
data. 
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PRODUCT ACCURACY 
AND TIMELINESS 


Principle 

The accuracy of ETS's principal products and the timeliness with which they are 
made available are important parts of the responsibility ETS has undertaken 
with respect to its sponsors and the diverse public it serves. 


Policies 

A. ETS will establish standards of accuracy and timeliness with respect to each 
principal product. 

B. ETS will use quality controls that are adequate to assure that its standards of 
accuracy and timeliness are met. 

C. ETS will make realistic delivery commitments and reasonable efforts to meet 
those commitments. 

D. ETS will sacrifice the timeliness of the delivery of information if the desired 
accuracy of that information is substantially in question. 

E. ETS will seek to inform those adversely affected if, subsequent to its release, 
information has been found not to meet ETS standards of accuracy. 

F. ETS will seek to inform those adversely affected if there is a probability that 
there will be substantial departure from ETS standards of timeliness with 
respect to a principal product. 


Procedural Guidelines 

1. Principal products should be identified and a standard of accuracy using units 
of measurement appropriate to the type of product should be established for 
each. 

2. When appropriate, quality control should include an adequate and indepen- 
dent recomputation and a visual reexamination of ETS-processed information 
based on an appropriate sample of cases sufficient to identify errors within the 
limits of the applicable standards of accuracy. 
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3. When the computational nature of the information is such that it is impossible 
or impractical to determine the accuracy of the information by independent 
recomputation, staff members who are technically competent to do so should 
assess its "reasonableness" as a part of quality control. 

4. There should be a quality inspection of intermediate products when: 

a. the accuracy of variable information (e.g., parameter data, algorithms), 
verified by independent recomputation or assessment, influences the 
nature of an ets process or computation and is critical to the process that 
generates an ets principal product; or 

b. detection and correction of errors would facilitate meeting the delivery 
commitment on the principal product; or 

c. the nature of the principal product is such that it is impossible or imprac- 
tical to determine the accuracy of the information by independent compu- 
tation using the source data. 

5. Quality control for principal products that do not coniain processed informa- 
tion (such as bulletins of information or test books) should include inspection 
of a sample prior to release of the product. If the product is released from an 
outside vendor (e g., outside publisher) or a sponsor's agent, quality control 
should include inspection of those components of the principal product that 
contain critical information on ETS-provided services. 

6. Quality control of information given in letter or telephone responses should 
include a periodic audit of a sample. 

7. Failure to meet standards of accuracy and timeliness should be reported to a 
designated ETS staff member for resolution. 

8. A principal product that does not meet established standards of accuracy 
should not be released until appropriate corrective action is taken, unless 
release would be for the benefit of the score recipient and users and permis- 
sion to release is given by the cognizant ETS officer. 

9. If an error is found in critical information already released by ETS, the correct 
information should be promptly distributed. 

10. Process control methods (e.g., a predefined schedule including a delivery date 
and contingency procedures for dealing with volume surge) should be estab- 
lished for the production of each principal product to help assure its delivery 
by the scheduled delivery date. 

11. If it is likely that there will be a substantial departure from ETS standards of 
timeliness with respect to a principal product, those who would be adversely 
affected should be so notified. 
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RESEARCH AND DEVELOPMENT 


Principle 

A continuing program of research and development conducted in compliance 
with professional standards with respect to quality and ethical procedures is 
necessary to maintain the high quality and social utility of ETS contributions to 
education. This includes basic inquiry to increase understanding of educa- 
tional processes and human development ; evaluative and applied research in 
response to the needs of the educational community ; and research and devel- 
opment to improve ETS products and services. Publication of the results of sig- 
nificant ETS research is of benefit to ETS and the profession because it permits 
others to use , build upon or improve ETS work. 


Policies 

A. ETS will devote appropriate research efforts to improving education through 
the discovery and conceptual integration of new principles and understand- 
ing. This research will be aimed at extending knowledge of the learner and 
learning processes, of learning environments and educational treatments, of 
educational institutions and of the interacting factors that influence human 
development. 

B. ETS will devote appropriate research efforts to the improvement of the tech- 
nical quality of ETS products and services. Among the important issues 
addressed by this research will be problems of test development, reliability, 
equating, validity, and meaningfulness of interpretation. 

C. ETS will devote appropriate research and development efforts to the identifica- 
tion of needs of the educational community and to the creation, improvement 
and evaluation of instruments, systems and programs of service that meet 
these needs. 

D. ETS will conduct its research under appropriate procedures that protect the 
rights of privacy and confidentiality of human subjects or respondents. 

E. ETS will follow procedures to insure that ETS research is of high quality. Stan- 
dards of quality in research refer to such matters as the identification of rele- 
vant data, the choice of suitable methods of collecting and analyzing data, the 
logic and objectivity of analysis and interpretation, the exploration of relation- 
ships between research problems and findings, on the one hand, and existing 
knowledge, theories and methodologies on the other, and the thoroughness 
and care of project planning and management. 


Approved For Release 2001/11/15 : CIA-RDP00-01458R0001001 10006-4 



Approved For Release 2001/11/15 : CIA-RDP00-01458R0001 001 10006-4 


F. ets will undertake research only if its potential benefits outweigh the inconve- 
niences of or risks to the subjects or respondents who are involved. 

G. ets will encourage the dissemination of full accounts of ets research in the 
usual professional forums and will provide internal means by which the results 
of ETS research can be published. 

Procedural Guidelines 

1. lo maintain the quality of operational programs, ets should engage in the fol- 
lowing activities: 

a. study and research on the test development process, including systematic 
development and evaluation of new item types and approaches: 

b. studies to determine the sources of significant differential performance of 
sex, ethnic, handicapped, and other relevant subgroups on ETS tests; 

c. periodic evaluation of current approaches to aptitude and achievement 
measurement to determine fairness, validity and appropriateness for signif- 
icant subgroups such as minorities and women; 

d. research related to reliability theory and practice, including methods of 
determining the reliability of classification decisions; 

e. study of the equating methods presently in use and development of 
improved methods as limitations in the applicability of the present 
methods are observed; and 

t. research to advance measurement techniques and selection and classifica- 
tion models relevant to fairness and validity. 

2. Research projects should be undertaken in such areas as learning and cogni- 
tion, personality and social influence, teacher behavior and instructional pro- 
cesses, socialization and human development, and the economics and sociol- 
ogy of education as a means of improving educational policies and practices. 

3. Efforts should be made to develop instruments and programs of service in 
areas such as measurement, institutional and program assessment and evalua- 
tion, instruction, guidance, financial aid, certification and licensing, and tech- 
nology that would be of educational and social utility. 

4. Proposals for research to be conducted by ETS and involving human subjects 
or respondents should be considered by the Committee on Prior Review of 
Research, under its procedures for review, to verify that proper arrangements 
have been made for protection of the welfare and rights of human subjects. 
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5. Researchers should not conduct research projects without the consent of sub- 
jects and respondents. In the case of young children, the consent of parents or 
a legal guardian, or of appropriate institutional representatives, should be 
obtained. 

6. Each research proposal should be reviewed by one or more persons who are 
competent in the field within which the proposal falls. They should be satisfied 
that professional standards of quality and ethical conduct are met. 

7. Identifiable data should be released from ets to researchers other than those 
who originally conducted the research only when one of two conditions have 
been met: 

a. Consent to do so has been given by or on behalf of the subjects or respon- 
dents or by those who have given consent on their behalf; or 

b. the Committee on Prior Review of Research has determined that release of 
the data serves a public need, that there is no satisfactory and reasonable 
alternative way of obtaining the information, that the recipient researcher 
will use the data in appropriate ways and that there are adequate assur- 
ances of confidentiality. 

8. After the data-col lection phase of a research project has been completed, sub- 
jects should not be expected to provide additional data for a follow-up study 
unless such participation was part of their original agreement to serve as sub- 
jects, or their consent for follow-up is obtained or the follow-up study has 
been approved by the ETS Committee on Prior Review of Research. 

9. The results of measures of performance based on experimental situations or 
tests, the interpretation of which is, therefore, tentative, and whose applied 
use is not yet supportable, should not be reported to subjects, or to the insti- 
tutions providing the subjects, unless there is relatively little danger of misin- 
terpretation or misuse of the information that would be harmful to those 
individuals or institutions, or unless the use is part of a feasibility study or 
experimental condition. Stipulations regarding nonissuance of such reports 
should be made to participants in advance of the data collection. 

10. The results of each research project undertaken with respect to a particular ETS 
program or service should be available for dissemination unless a specific 
need to restrict publication to protect confidentiality or for other program pur- 
poses is identified prior to the beginning of the project and made known to 
the appropriate individuals. 

11. The contracts under which research is undertaken for agencies or institutions 
outside ETS should permit publication of the results of the research unless a 
specific need to protect the research results is identified prior to the beginning 
of the research and made known to the appropriate individuals. 
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TESTS AND MEASUREMENT- 
TECHNICAL QUALITY OF TESTS 


This section which deals with ets testing activities is divided into seven subsec- 
tions that are devoted to test development, test administration, reliability, 
scale definition, equating, score interpretation, and validity. 

Principle 

High standards of quality and fairness in constructing, administering, reporting, 
interpreting and evaluating FTS tests are central to ETS's capability to function 
effectively as an educational service and research organization. 

Policies 

A. its will strive to develop tests in which the attributes measured, procedures 
lollowed, and criteria used will be unbiased with regard to a heterogeneous 
group of examinees and appropriate to the use for which the test is designed. 

B. ns will establish standards for test-administration processes that minimize 
variations in test performance due to circumstances or conditions not relevant 
to the attributes being measured. 

C. Eis will establish for its tests a high degree of reliability (accuracy of measure- 
ment), consistent with the requirements and the purposes of the test. 

D. eis will develop scales for reporting scores in a rational fashion, consistent 
with the requirements and the purposes of the test. 

E. eis will provide equating systems, when appropriate, for the perpetuation of 
st ales for reporting scores at the highest level of precision practicable. 

F. Eis will make available to sponsors, institutional or agency users, and exam- 
inees data for interpreting scores on ETS tests that foster appropriate use of 
those scores. 

C. Recognizing that test validation is a responsibility of both test users and test 
developers, ets will encourage and assist test users in their validation efforts 
and will itself make available tests that are designed to meet professionally 
acceptable standards of validity provided the use of such tests is consistent 
with the primary purposes for which the tests were developed. 

H. ets will adhere to appropriate professional standards, such as those published 
in Standards of Educational and Psychological Tests and Principles for the Vali- 
dation and Use of Personnel Selection Procedures. 
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Procedural Guidelines: Test Development 

1. Policy and substantive contributions to the test development process should 
be obtained from qualified men and women who are not on the full-time staff 
of ets and who are drawn from diverse backgrounds and appropriate special- 
ties within professional fields (e.g,, various kinds of institutions and programs, 
relevant philosophies and points of view, and major ethnic, handicapped and 
other relevant subgroups of the population). 

2. Appropriate background information for use in the development of a test 
should be documented at appropriate stages in the development process and 
include: 

a. the purpose for which the test is intended to be used; 

b. the nature of the population that will take the test; 

c. the relevant procedural, financial or time constraints that will influence the 
available test development methods and their likely outcomes; 

d. for achievement tests, the kinds of curricula for which the test is designed; 
and 

e. for job-related tests, the elements in training or employment that are 
related to performance on the job. 

3. For each test, specifications should be developed and reviewed by a process 
that provides information from the following perspectives: 

a. Content and Skills— specifications should include the psychological, educa- 
tional, or other domains to be sampled; the relative weight to be given to 
each domain; the appropriate level of proficiency to be required within 
each domain; a balance with respect to curricular differences. 

b. Test and Item Format-specifications should include the item (question) 
types that are most clearly related to content or skills to be measured; the 
appropriate level of language or reading; requirements regarding clear and 
comprehensive directions and sample items or the need for a sample test; 
and whether free-response, multiple-choice or other machine-scorable for- 
mats can be used. 

c. Psychometric— specifications should include the level of difficulty of the 
test, the distribution of item difficulties (when pretested items are used); 
guidelines for evaluating the homogeneity among items within a test and 
the relationship between subtests or tests; equating requirements; number 
of items and time allotted. 

d. Sensitivity-specifications for tests should require material reflecting the 
cultural background and contributions of women, minorities, and other 
subgroups; specifications should also require a balance of positive conno- 
tations if negative connotations are made in any references to these 
groups. 
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4. Except for tests designed to measure rate of performance, the number 
of items in a test that has a specified time limit should be chosen so that time 
is not a decisive factor in performance, at least for the large majority of 
esaminees. 

5. Subject matter and measurement specialists familiar with the purpose of the 
test and with the characteristics of the intended population should review the 
test items for accuracy, content appropriateness and the adequacy with which 
the items sample the domain. 

b. The individual items in a test should meet appropriate technical standards 
such as those contained in the manuals for item writers used in the test devel- 
opment area. 

7. Individual test items, and the test as a whole, should be reviewed to eliminate 
language, symbols or content which are generally considered potentially 
offensive, inappropriate for major subgroups of the test-taking population or 
serving to perpetuate any negative attitude which may be conveyed toward 
these subgroups. No item in any test should include words, phrases or 
description that is generally regarded as biased, sexist or racist (e.g., demean- 
ing modifiers and stereotypes). 

8. I he items in a test should be reviewed by editorial specialists for clarity, accu- 
racy, consistency, and, when appropriate, for conformity with standard edi- 
torial style. 

9. tests should contain clear and complete directions. Enough sample problems 
should be provided in test-program publications so that the examinee can 

inderstand the nature of the task and the test-taking procedures. Where there 
is a need to provide a general orientation to testing, as when testing young 
children, practice tests-included either in descriptive material or at the time of 
lest administration— should be used. 

10. I he typography, directions, and arrangement of items in the test booklet 
should facilitate the task of test takers. When appropriate, tests should be 
made available to handicapped individuals, such as sight-deficient candidates, 
ihrough tapes, readers or special printing. 

11 . Methods should be employed to evaluate the appropriateness of items before 
(heir operational use in a program or before the reporting ot scores. Appropri- 
ate methods include pretesting, preliminary item analysis (using the first oper- 
ational use of items as an opportunity to identify inadequate items) or careful 
review of the results of administering similar items to a similar population. In 
assessing the appropriateness of items before their operational use, efforts 
should be made to include representative samples of the operational test- 
taking population. 
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12. The operational use of each test should be followed by systematic item anal- 
yses using appropriate criteria and by test analyses. These analyses should 
include reliability, intercorrelations of sections or parts, and speededness. 

13. Studies relating item performance to subgroups should be carried out for new 
or substantially revised tests when there are adequate data concerning suffi- 
cient samples of large subgroups whose education and experience may be dif- 
ferent from the majority of examinees. 

14. The specifications for tests in ongoing programs should be reviewed for rele- 
vance and appropriateness before each new form is created, ets staff and 
adviseis should consider whether changes in the field, discipline or curricula 
require a revision of the specifications. 

15. When major changes are made in test specifications, consideration should be 
given to the implications of such changes for score comparability and whether 
it is necessary to change the test name or otherwise communicate to those 
who interpret test scores that comparisons with earlier tests may be inap- 
propriate. 

16. When test forms are used for a number of years in a program, they should be 
reviewed periodically for their appropriateness. The frequency of such review 
should be determined by the amount of change occurring in the population of 
test takers or the subject matter domain. Test forms that are found to be out- 
dated should be revised or withdrawn from use. 
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Procedural Guidelines: Test Administration 

1 . Information should be made available to prospective examinees and (in some 
programs, to parents or guardians as well) in advance of the test administra 
tion with respect to the following, as appropriate: 

a. the purpose of the test and what it measures; 

b. the nature of the test items (including samples of typical item types); 

c. the relevant instructions for taking the test, including instructions for 
guessing, changing answers, and strategy involving speed and accuracy in 
taking the test; 

d. identification requirements and the consequences of not having identi- 
fication; 

e. the consequences of misconduct by the test taker; 

f. background and experience relevant to test performance; 

g. the location of test centers, the test dates and special testing arrange- 
ments that can be made; 

h. the procedures for registering for the test and changing the centers, 

i. the structure of test fees and fee waivers; 

I. special arrangements available for administering tests to handicapped 
individuals; 

k. the reporting of scores; 

i. procedures for canceling test scores by the candidate and reasons why ETS 
or the sponsor of the test might cancel scores; and 

m. the procedures for registering complaints. 

2. Program publications should be reviewed for language or descriptions gener- 
ally regarded as biased and offensive. For example, the exclusive use of mas- 
culine pronouns should be avoided as should the implication that all persons 
in a given category (for instance, examinees, supervisors, counselors, or teach- 
ers) are either females or males (unless, of course, the category is logically 
restricted to members of a single sex). Illustrations, examples and practice 
items in test-information publications should represent males, females, minor- 
ity and majority groups, and individuals in ways that indicate respect and 
awareness of valuable contributions. 

3. 1 he facilities at which tests are administered should be places that are conve- 
nient for the majority of examinees, nonsegregated and comfortable. At ieast 
portions of those facilities should be accessible to and responsive to the needs 
of handicapped individuals. 
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4. ets should enlist test center supervisors and staff with demonstrated sensitivity 
to the anticipated sex and ethnic composition of the examinee group, based 
on prior experience. When appropriate, persons affiliated with institutions 
attended by significant numbers of those examinees should be included. 
Minority group supervisors and/or proctors should be employed, and test sites 
should be located in minority communities whenever appropriate and 
feasible. 

5. Test center supervisors and staff should be familiar with the procedures for 
administering a standardized test and should be provided with a description of 
the testing program, a description of the candidate population, and specific 
instructions for administering the test. Instructions concern such subjects as 
the duties of test supervisors, associate supervisors, and proctors; the receipt, 
storage and return of test supplies; the admittance of examinees to the testing 
rooms; the distribution of test materials; procedures to be followed in admin- 
istering tests to handicapped individuals; procedures to be followed in in- 
stances of suspected cheating; procedures to be followed in other cases of 
candidate misconduct; and procedures to be followed in case of emergency. 

6. Test performance can be affected by the psychological atmosphere of the test- 
ing center. Test supervisors should be informed of this and instructed to take 
measures to avoid an adverse situation. For example, test supervisors should 
be instructed, when it is appropriate and feasible, to have minority- as well as 
majority-group persons, women as well as men, read test directions and to 
recognize questions from examinees following an impartial procedure. 

7. ETS should provide the test center supervisor with directions to be read aloud 
to examinees before the test begins. These directions should include informa- 
tion relating to: procedures for marking answer sheets, timing of test sections, 
strategies for guessing, time and duration of test breaks and examinees' use of 
unauthorized aids. Test supervisors should check to see that examinees under- 
stand their task and the procedures to be followed. 

8. Reasonable efforts should be made to eliminate opportunities for examinees 
to attain scores by fraudulent means by stipulating requirements for identifi- 
cation, assigning examinees to seats and requiring appropriate space between 
seats, 

9. Appropriate procedures should be applied after the test administration to 
identify scores of questionable authenticity, to resolve issues of authenticity 
and to provide for prompt reporting of questioned scores found to be 
authentic. 

10. A systematic program for observing test administrations should be conducted 
by trained ETS staff members or other qualified individuals: to review the test- 
ing procedures with the test supervisors, to ensure appropriate testing condi- 
tions, to ensure adequate maintenance of test security at the test centers and 
to relay questions and concerns from the field to the appropriate ETS office. 
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11. lesting programs should have detailed procedures for investigating and 
resolving examinees' complaints of irregular test administration or score 
reporting. 

12. Comments and suggestions should be solicited from supervisors by such 
means as the Supervisor's Comment Sheet and meetings of supervisors to pro- 
vide ets staff with information to improve future administrations. 

13. Supervisors should be required to record and report to ETS information on 
irregularities (such as mistiming, defective materials, power failures and cheat- 
ing) so that ets can evaluate the possible effect of such occurrences on exam- 
inees' test performance. 

14. An individual who has taken a test should be provided information that will be 
helpful in interpreting scores on that test. 
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Procedural Guidelines: Test Reliability 

1. When test scores are reported to institutional or agency users or to individual 
examinees, information about the reliability of the test should be documented 
and should include: 

a. a reliability coefficient and an overall standard error of measurement (sev- 
eral indices may be provided if more than one method of assessing reliabil- 
ity has been used; alternate-form information should always be provided if 
available); 

b. standard errors of measurement for score regions if decisions about indi- 
viduals are made in those score regions and if the overall regions and the 
overall standard error are judged inappropriate; 

c. the formula(s) used to estimate reliability and/or appropriate references; 

d. a justification of the method(s) used to assess reliability; 

e. a specification of the major sources of measurement error accounted for in 
the reliability analysis; 

f. a specification of the time interval between testings if alternate-form or 
test-retest reliability is used; 

g. the number of observations, the mean and standard deviation of the analy- 
sis sample (ranges or averages are acceptable in cases where the reliability 
information is derived from several samples); 

h. speededness data; and 

i. correlations of subscores within the same test or battery of which the test 
is a part. 

2. If reporting any of the reliability information required under Guideline 1 is 
inappropriate, the reasons should be stated in appropriate program docu- 
ments and, if possible, alternate information about consistency should be 
provided. 

3. Efforts should be made to provide reliability information in an appropriate 
form to the examinees to whom the scores are reported. 

4. The method(s) used for assessing reliability should: 

a. take into account the most common sources of error generally considered 
significant for test interpretation (e.g., guessing, instability over time, item 
and content variation, and rater inconsistency); and 

b. be appropriate to the nature of the test, in order not to seriously over- or 
underestimate reliability. 
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Procedural Guidelines: Scale Definition 

1. Raw scores on a test or subtest (including percentages of questions answered 
correctly) should not be reported by ETS for individual examinees or in sum- 
mary form for groups of examinees except under either of the following cir- 
cumstances: 

a. when it is anticipated that only one edition of the test will be offered for 
use in the foreseeable future or it is demonstrated by appropriate empirical 
procedures that raw scores on all the editions to be compared are inter- 
changeable; or when raw scores on that test edition will not be compared 
directly with raw scores on another test edition; or 

b. when reported in conjunction with a scaled score and in a context that sup- 
ports appropriate interpretation, such as when a copy of the test itself is 
available or when individual or group responses to individual items, 
depending on whether individual or group performance is being assessed, 
are available. 

2. It a test or test battery yields multiple scores for an individual and scaled 
scores are to be used directly (e.g., without reference to norms tables) in inter- 
preting performance profiles, the scales should be normatively defined and 
each should be defined with respect to the same population. 

3. When different tests in a program are taken by different examinees whose 
scores are to be directly compared, the scales for the tests should take into 
account possible differences among the groups of examinees who take the 
various tests. 

4. Established scales should not be redefined except under compelling circum- 
stances. If a scale is to be substantially redefined, the numerical values should 
be changed substantially to minimize the possibility of confusion between test 
results expressed on the revised scale and results expressed on the original 
scale. An exception to this guideline may appropriately occur if the test in 
question is one of a set of tests for which a single range of numerical values 
(e.g., 2080) is used and the scales for other tests in the set have not been 
redefined. 

5. Scale properties that affect score interpretation and use should be described 
in program publications available to the examinees and to institutional or 
agency users. 

6. Technical manuals and interpretive publications for institutional or agency 
score users and examinees should indicate, in language appropriate to the 
audience, whether a distributively based scale is intended to be normative or 
nonnormative. If it is intended to be normative, the group should be described. 

7. Whenever a normatively defined scale no longer conveys useful normative 
information, all published descriptions of the scale should be changed 
accordingly. 
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8. Program publications should caution score recipients (users and examinees) 
that scores received on different tests that are reported on scales that are sim- 
ilar in appearance may not be equivalent. 

Guidelines 9 through 14 apply only to scales established after guidelines 1-8 were 
published on August 1, 1977. 

9. If a scale is to be distributive, the choice between a normative and nonnorma- 
tive distributive scale should take into account: 

a. the extent to which normative interpretation with reference to a particular 
population will be appropriate and useful for all examinees who take the 
test and for all purposes for which the scores are intended to be used; 

b. the probable time period during which the normative information con- 
veyed by the scores will continue to be descriptively appropriate; and 

c. the feasibility of identifying and testing a suitable group of examinees on 
which to base a normative scale. 

10. The choice between a distributive and nondistributive scale should take into 
account the use for which the test was intended and to which the test is likely 
to be put. 

11. If a scale is to be defined with reference to standards of performance, the basis 
for establishing the standards should be determined empirically or rationally 
rather than arbitrarily. 

12. The conventional grade- or age-equivalent score (the grade or age for which a 
particular score is the average) should not be used to establish the score scale 
for a test or system of tests. This type of score, as it typically has been derived, 
should be avoided altogether as a basis for reporting test performance. How- 
ever, the grade (or age) for which a particular scaled score on a test is the aver- 
age, referred to here as a "grade (age) level indicator" to distinguish it from the 
conventional grade-equivalent (age-equivalent) score, may be reported to 
help in score interpretation, if the practices customarily followed in deriving 
and presenting grade-equivalent (age-equivalent) scores are modified in accor- 
dance with ETS criteria that obviate the technical interpretive problems that 
grade-equivalent (age-equivalent) scores create. 

1 3. The choice of a scale should take into account the likelihood of confusion with 
other widely used scales. 

14. In establishing the number of distinct scale values to be reported, considera- 
tion should be given to the relative importance of the need to avoid errone- 
ous distinctions among individuals (by reporting different scores for individuals 
whose true scores are the same) and the need to maintain distinctions that, on 
the average, will be correct (by reporting different scores for groups of individ- 
uals whose average true scores are different). 
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Procedural Guidelines: Equating 

1. Adequate equating should precede comparisons of the test performance of 
two or more individuals or groups on nonidentical items or sets of items such 
as test offerings in which successive, or alternate, forms are used inter- 
changeably. 

2. Statistical methods selected for equating should be used only under circum- 
stances that are consistent with the assumptions under which the methods 
have been developed. 

3. In regular and continuing testing programs that are available to users, inte- 
grated, long-range systems of equating the scores on all successive editions of 
the test should be used and described in technical publications. 

4. For those tests that are offered for institutional use (as distinguished from 
externally administered tests offered in testing programs) of which only a 
limited number of forms are available, equating of new forms should be based 
on specially designed studies in which examinees or groups of examinees are 
selected by an appropriate sampling procedure to take the alternate forms or 
alternate sequences of forms. 

5. When test forms are equated with the use of common (anchor) items, the psy- 
chological task of taking those items (represented, for example, by the direc- 
tions, the context of the items and the speededness of the part of the test in 
which the items appear) should be the same for all examinees. 

6. When the common items used for equating are not representative of the tests 
being equated, the groups of examinees used for equating should be as nearly 
as possible equivalent. 

7. In the continuing testing programs, statistical checks (e.g., check equating, 
special scale-stability studies) should be employed to permit regular assess- 
ment of the precision of the equating. 
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Procedural Guidelines: Score Interpretation 

1. Effective test use and meaningful score interpretation should be supported 
and augmented by: 

a. the development of appropriate test norms based on administering tests to 
samples from a defined population when there is a reasonable expectation 
that a large proportion of the schools or other units selected for the norms 
sample will agree to participate; or 

b. a rationally developed system of interpretation shared with score recipi- 
ents when score interpretation is not developed from normative data. 

2. Tests offered for sale and described by ETS as standardized tests (as distin- 
guished from tests offered in testing programs) should have adequate norms 
or other information for use in interpreting test results. 

3. When test norms are developed by administering tests to samples from a 
defined population, the resulting norms should be representative of any rele- 
vant subgroup, including those defined by sex or ethnicity, in proportion to 
their frequency in the defined population. Such subgroups may be deliber- 
ately over-sampled for more precise estimation of the statistical characteristics 
of the population by procedures that take over-sampling into account. Data 
on the proportions in the sample and in the population, when available, 
should be reported in an appropriate technical publication. 

4. The report of a special norms study should provide information on: 

a. the sampling design; 

b. the participation rate of institutions or individual respondents in the sample, 

c. characteristics of the participating institutions and individuals; 

d. weighting systems used in preparing norms; and 

e. estimates of sampling variability along with an acknowledgment, when 
necessary, that such estimates do not take into account biases arising from 
nonparticipation. 

5. When descriptive statistics based on program testing (as distinguished from 
norms based on special norms studies) are published, the following guidelines 
should be used: 

a. both table titles and descriptive material should make it clear that the sta- 
tistics are based on examinees or participating institutions or other using 
agencies; 

b. the descriptive material should define the nature of the group by identify- 
ing the appropriateness of the sample and the factors that relate the back- 
ground of the group to test performance, and by acknowledging explicitly 
that the sample is self-selected; 
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c. when possible, reports should be prepared to show comparisons of data 
based on program examinees or institutional characteristics with relevant 
data on variables from other sources; and 

d. when information about interpretive data is prepared for different user 
groups, the presentation, whenever practicable, should be adapted to the 
needs and background of each group. 

6. When norms are developed from program testing, the age, sex and ethnic 
composition of the program norms group should be described whenever such 
information about subgroup membership is available 

7. In testing programs, descriptive statistics should be compiled periodically 
from a sample or entire population in order to monitor the participation and 
performance of males and females drawn from diverse backgrounds, interests 
and experience (e.g., major ethnic group, handicapped status and other rele- 
vant subgroups of the population of interest). 

8. If norms intended for use in the interpretation of individual scores are pre- 
sented separately for males and females or for members of specific ethnic 
groups, the rationale should be carefully described. Separate norms may be 
justified for scores used primarily for guidance when access to the experiences 
needed to earn a high score is clearly related to subgroup membership and a 
more direct index of access is not available. The existence of score differences 
between subgroups does not in itself justify presentation of separate norms. 

9. Descriptive statistics prepared separately for subgroups of the relevant test- 
taking population, but not intended for use in interpreting individual scores, 
should not be presented in a way that encourages their use for such a purpose. 

10. Institutional or agency users and examinees should be informed of the stan- 
dard error of measurement of a score, and test interpretation materials should 
point out the limitations of test scores and encourage score users to take into 
account the possible scores a test taker might achieve on retesting. 

11. Statistical data used in score interpretation should be revised annually except 
when less frequent revision is judged to be appropriate as, for example, when 
norms are based on special studies. A statement of the period in which the 
data were collected should be included in any publication that presents the 
data. 

12. Institutional or agency score recipients should be provided with interpretive 
materials designed to be helpful for using scores in conjunction with other 
information, setting cutting scores where appropriate, interpreting the scores 
for special subgroups (e.g., ethnic minorities, males, females, and handi- 
capped students), conducting local normative studies, and developing local 
interpretative materials. 
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Procedural Guidelines: Test Validity 

1. ETS should provide evidence of the validity of its tests in relation to the princi- 
pal purposes or intended uses of the tests. One or more of the following may 
be applicable: 

a. When test scores are to be interpreted in terms of degree of mastery of the 
knowledge, skills, or abilities of a domain represented by the test, content 
validation evidence should be provided. 

b. When test scores are to be interpreted in terms of the prediction of future 
behavior, criterion-related validation evidence should be provided. 

c. When test scores are to be interpreted as a measure of a theoretical con- 
struct, construct validation evidence should be provided. 

2. Evidence of content validity should be based (a) on a careful determination 
and analysis of the domain(s) of interest and of the relative importance of top- 
ics within the domain, and (b) on a demonstration that the test is an appropri- 
ate sample of the knowledge or behavior in the domain(s). A report on evi- 
dence of content validity should present descriptions of the procedures 
employed in the study, including the number and qualifications of experts in- 
volved in the analysis of the domain or evaluation of the relevance and appro- 
priateness of the test. 

3. Construct validation should be based on: rational and empirical analyses of 
processes underlying performance on the test in question including, where 
appropriate, noncognitive as well as cognitive functions. Empirical evidence 
relevant to the analyses should include results of investigations of the degree 
to which test scores are related or unrelated to other variables in ways implied 
by intended interpretations. 

4. Criterion-related validation should be used only when technically sound and 
relevant criteria are available or can be developed and when other conditions 
affecting feasibility warrant the study. 

a. Criterion-related validation should involve as many performance variables 
as necessary to permit evaluation of the effectiveness of test scores for pre- 
dicting the types of behavior they are intended to measure. 

b. Criterion-related validation should not combine variables to form a single 
criterion measure unless such a procedure is justified by logical considera- 
tions or empirical evidence or the practical requirements of the intended 
use of the results. 

c. Criterion data should be collected in a way that permits an assessment of 
the reliability of each criterion variable, but with the understanding that 
there may be several sources of irrelevant variation (e.g., sampling of crite- 
rion content, source of criterion ratings or data, and so forth). 
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5. Interpretations of correlations between test scores and criterion variables 
should take into account such factors as sample size, criterion reliability, pos- 
sible restriction in the range of scores obtained in the validity study sample, 
and other contextual factors. 

6. The method(s) by which any validation is accomplished should be fully docu- 
mented; such documentation should include appropriate details such as the 
nature and reliability of the criteria, a description of the subjects used, the 
materials surveyed and the qualifications of the experts who made judgments 
regarding the appropriateness and importance of test content. 

7. Where adequate methods are employed to insure equivalence of scores on 
alternate forms, it is not necessary that each new form be validated. New vali- 
dation studies should be made if revised tests have substantial changes, such 
as different item types, or if they sample a revised performance domain. 

8. When appropriate and feasible, the validity of a test should be investigated 
separately for subsamples of the test-taking population. 

9. When a name of a test is established, it should not imply more than is justified 
by evidence of validity. 

10. Information should be made available to institutional and agency users that 
would be of assistance to them in planning and conducting local validity 
studies. 
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TEST USE 

Principle 

Proper and fair use of ETS tests is essential to the social utility and professional 

acceptance of ETS work. 

Policies 

A. ETS will set forth dearly to sponsors, institutional or agency users, and exam- 
inees the principles of proper use of tests and interpretation of test results. 

b. ETS will establish procedures by which fair and appropriate test use can be pro- 
moted and misuse can be discouraged or eliminated. 

Procedural Guidelines 

1. Program publications should: 

a. describe appropriate uses and caution against potential misuses of pro- 
gram tests; 

b. explain clearly that test scores reflect past opportunity to learn and dis- 
courage test interpretations that go beyond reasonable inferences from 
test performance; 

c. emphasize that an individual's test score should be interpreted in the con- 
text of other information about him or her; 

d. provide appropriate information about test content, difficulty, and purpose 
to help the institutional or agency user select instruments that meet the 
measurement requirements of the situation and avoid selecting, requiring 
or using inappropriate tests; 

e. invite institutional or agency users to consult with the program sponsor 
and/or ETS about their current or intended uses of ETS-developed tests and 
identify the offices to be contacted for this purpose; 

f. summarize results of research relevant to the use of the test or cite refer- 
ences in which such results are reported; 

g. describe adequately and clearly scale properties that affect score interpre- 
tation and use; 

h. advise institutional or agency users that decisions about the application of 
single or multiple prediction equations, based on distinguishing character- 
istics such as sex, ethnic group or curricular emphasis or training, should be 
preceded by careful examination of social, educational and psychometric 
factors; 
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i. advise institutional or agency users that if examinee grouping based on test 
scores is practiced, provision should be made for frequent review of group 
assignments to determine actual performance; 

j. stress that pass-fail or cut-off scores established for such purposes as 
admission, credit, or certification should be used as a basis for decision- 
making only if the institutional or agency user has a carefully developed 
rationale, justification, or explanation of the cutting score that is adopted; 
and 

k. encourage institutional or agency users to reexamine cut-off score policies 
periodically to minimize or eliminate possible disproportionate exclusion 
of members of any group such as men and women drawn from diverse 
backgrounds (e.g., major ethnic, handicapped and other subgroups of the 
population of interest) in the face of other evidence that would predict 
their success or indicate their competence. 

2. Special (nonprogram) publications should be developed and disseminated by 
ETS to promote fair use of tests and discourage misuse of tests. 

3. C omplaints or information about questionable interpretation or use of 
reported scores should be investigated by means of procedures designed for 
detecting misuse. Such procedures should be documented, and records 
should be kept of such complaints and their disposition. 

4. In cases where a clear misuse is brought to its attention, ETS should inform the 
sponsor and the institutional or agency user of ETS's opinion as to the misuse 
and seek voluntary correction of the misuse. If reasonable efforts to seek 
voluntary correction are not successful, ETS, in conjunction with the sponsor, 
should take steps to determine whether to continue supplying tests or report- 
ing scores to the institutional or agency user. 
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TECH N ICAL ASSISTANCE 
ADVICE, AND INSTRUCTION 


Principle 


ETS is dedicated not only to providing measurement programs and conducting 
research, but also to promoting increased understanding of measurement and 
test use. 

Policies 

A. ETS will develop and offer instructional programs in the areas of measurement, 
evaluation, and related research through such forms as publications, seminars, 
in-service training, intensive residence courses, workshops, internships and 
conferences. ETS may undertake these activities independently or in coopera- 
tion with other agencies, professional groups or educational institutions. 

B. ets will provide advice and information on measurement-related issues and 
about ETS programs, research and services. In this activity, ETS will work, where 
feasible, in collaboration with other professional organizations that show a 
concern about measurement. 

C. ets will respond promptly to requests for advice, instruction and technical 
assistance related both to programs and services offered by ets and to the 
related areas of educational measurement, evaluation and research. 

D. ETS will conform to high standards of accuracy and professionalism in its advi- 
sory, instructional and technical assistance activities. 

E. ETS will provide advice, instruction and technical assistance to clients from the 
private and public sectors and from foreign and domestic government agen- 
cies to the extent that such services are consistent with ETS areas of expertise, 
meet accepted professional and ethical standards, and reflect an understand- 
ing of and respect for cultural differences. 

F. ETS will endeavor to promote increased understanding of the purposes and 
procedures of testing among professional groups and in the public sector; ETS 
will make this effort both independently and in cooperation with other organi- 
zations that share this responsibility. 
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Procedural Guidelines 

1. ETS's offices should offer advice, instruction and technical assistance; the staff- 
ing for such services should be determined by the nature of the services and 
the expertise required. 

2. The special requirements of audiences with varying needs, interests, cultural 
backgrounds and levels of knowledge should be considered when ETS pro- 
vides technical assistance, advice, or instruction. 

3. New developments in research or testing should be considered when tech- 
nical assistance, advice and instruction are offered. 

4. Technical assistance, advice and instruction offered to institutions or agencies 
should include guidance on how to use other information about examinees 
(such as previous academic performance, English as a second language, and 
family or cultural background factors) in conjunction with test scores. 

5. Comprehensive collections of reference materials relating to tests, measure- 
ment, evaluation and related research should be developed, maintained and 
made available to all ets staff members and, when appropriate, to professional 
groups and individuals outside the organization. 
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GLOSSARY OF TERMS 


Accuracy: The extent to which a principal product conforms to its specifi- 
cations or correctly reflects the source data within the specified limits of reli- 
ability. 

Client: (See Sponsor) 

Consent: Permission granted by an individual or that individual's parent or 
guardian to the use or release of data held by ETS; such permission granted 
upon receipt of a reasonable explanation of the purpose of the use or release 
and a reasonable explanation of the manner in which the results will be 
reported. 

Critical Information: Information that will be used to draw important infer- 
ences (a) about the sponsor, ETS-appointed external committees, institutional 
or agency user, examinee, subject or respondent, or (b) by the sponsor, insti- 
tutional or agency user, examinee, subject or respondent and which, if incor- 
rect, could be harmful. 

Distributive Scale: A scale that is defined to yield either a specified score dis- 
tribution or a specified mean and standard deviation for a particular group of 
examinees. 

ETS Board of Trustees: The ETS Board of Trustees is the governing body of ETS. 
There are 17 trustees. Sixteen are elected for four-year terms. New members 
of the Board are elected by current trustees. The President of ETS is an ex 
officio member. 

ETS-held Program Data Files: Information about individuals and institutions 
held by ETS and derived from ETS-provided services of collection, processing, 
storage, retrieval and dissemination. 

ETS-held Research Files: Information held by ETS and generated through 
ETS-conducted research intended to result in the development of new or im- 
proved techniques and materials for application in such areas as classroom 
instruction, evaluation of progress toward educational goals, counseling of 
students, and decision-making of school administrators. 
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Examinee: An individual who takes a test, developed and or administered by 
ETS. 

Institutional or Agency User: An organizational recipient of ETS-processed or 
produced information. 

Intermediate Product: Materials that are not released externally, but that are 
necessary to the production of the principal product. 

Nondistributive Scale: A scale that is defined without reference to the ob- 
served test performance of a particular group. 

Nonnormative Scale: A scale that is based on the performance of any conveni- 
ently available subgroups of examinees for whom the test is appropriate. A 
score on a nonnormative scale is not intended to convey information about 
an examinee's standing in relation to a defined population. 

Normative Scale: A scale that is based on the test performance of a sample of 
examinees, selected as prescribed by a specified design, from a clearly defined 
population. A score on a normative scale is intended to convey useful infor- 
mation about the performance of a particular examinee in relation to the per- 
formance of that population. 

Principal Product: ETS-produced or processed materials (e.g., annual reports, 
performance data, score reports and admissions tickets) that are released or 
transmitted to a sponsor, ETS-appointed external committee, institutional or 
agency user, examinee, subject or respondent, pursuant to a contract or pub- 
lished commitment. Standards with respect to accuracy and timeliness are 
applicable to principal products. 
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Principles for the Validation and Use of Personnel Selection Procedures, Divi- 
sion of Industrial-Organizational Psychology, American Psychological Associa- 
tion. Dayton, Ohio: The Industrial-Organizational Psychologist, 1975. 

Respondent: An individual who provides data to a research project in a man- 
ner and for a purpose different from either examinees or subjects. 

Sponsor: Educational, professional or occupational associations, federal, state 
or local agencies, public or private foundations which contract with ETS for its 
services. This category includes their governing boards, membership, and ap- 
pointed committees or staff. 

Standards for Educational and Psychological Tests , American Psychological 
Association (apa), American Educational Research Association, and National 
Council on Measurement in Education. Washington, D.C.: APA, 1974. 

Subgroup: A part of the larger population which is definable according to vari- 
ous criteria as appropriate, (e.g., by sex, race or ethnic origin, training or for- 
mal preparation, geographic location, income level, handicap and/or age). 

Subject: An individual who participates in an ETS laboratory or experimental 
research project. 

Testing Program: A set of arrangements under which examinees are scheduled 
to take a test under standardized conditions, the tests are supplied with 
instructions for giving and taking them, and arrangements are made for scor- 
ing the tests, reporting the scores, and providing interpretative information as 
part of a comprehensive ongoing service. A program is characterized by its 
continuing character and by the inclusiveness of the services provided. 

Timeliness: The degree to which a principal product is released or delivered to 
its recipient within a predefined schedule. 
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